things to do

[mflatt]

Some things that we can see currently need to be done, roughly in order of complexity:

• examples – See Example Rhombus/Shrubbery Programs #186. There's a chicken-and-egg problem here of having enough libraries and figuring out what the libraries should look like.

• fancy-app – Might be a relatively small detail, but better to sort out earlier instead of later.

• unit testing – Something small and simple enough to be in the core, and that composes well with abstractions like functions and modules (instead of having a parallel set of forms for abstracting over tests).

• errors – Figure out how to get primitives to complain with different names, but cheaply. Hopefully, this is not a compiler-improvement exercise.

• pretty printer for shrubberies – Sorawee has the starting point here.

• syntax-parse – Make more syntax-parse functionality available in the pattern and template sublanguages. One particular challenge is figuring out what needs to be in the base language and what can be in a separate library.

• Scribble – Adapted to shrubbery notation (shcribble?), with documentation forms that adapt better to different languages, and with a cleaned-up base library.

• in-source documentation – Possibly along the lines of scribble/srcdoc. Depends on Scribble adaptation.

• class system design – Build on the struct-like class form, but with methods? With racket/generic and/or racket/class integration?

• core datatypes – Maps, etc. Jack is working on this, but included here in an attempt at completeness and context.

• strings – Support for Unicode grapheme clusters (not just code units). There are questions of how Racket, port counting, and the drawing/GUI/editor libraries interact, so this is an especially wide-ranging problem. But there are some basic design decisions and functionality that might be worked out piecewise, and it might not be all that much code in the end.

• annotations/contracts – Extend the annotation library in the direction of contracts. Similar to syntax-parse, figuring out what needs to be core is part if the challenge here. This one interacts with research and ongoing work by Lukas @ Northwestern.

[97jaz]

@mflatt Regarding Unicode grapheme clusters: Racket uses data files from Unicode 7. The text segmentation spec has changed somewhat since then (mostly to account for emoji, I think, e.g., flags with zero-width joiners). What are your thoughts on moving to Unicode 14? (And do BC and CS need to change in lock step? I notice that they are both using Unicode 7, but I don't know if that was planned.)

[mflatt]

The read-char and read-byte versions are Racket loops that have to call into read-char and read-byte. Calling into non-inlined primitives is relatively slow on BC.

Meanwhile, read-bytes and read-string have all the counting work built in, which means staying in C on BC. The numbers where BC is 10x as fast are just wrong; I transcribed the wrong columns in a larger set of output, and those should be 2x. The ones where it's 2x or 3x as fast are because the C code is currently faster than the Scheme code. Partly, that speed is obtained in BC by just giving up when positions move out of fixnum range, while CS will keep counting. Partly, even though Chez Scheme generates fast code, I'm still able to even faster code via GCC/Clang for small-integer manipulation.

[samth]

Ok, that all makes sense. Can we get back some of the speed by assuming fixnums, at least on 64-bit platforms?

[mflatt]

Can we get back some of the speed by assuming fixnums, at least on 64-bit platforms?

It looks like I was wrong about fixnum aithmetic being relevant. Although I can make the generated code substantially simpler by using fixnum arithmetic, it doesn't run any faster (maybe because the CPU's branch predictor works).

[samth]

It might be worth testing that on x64 too, if you haven't.

[mflatt]

Some notes after further implementation at racket/racket#4350, racket/draw#49, and racket/gui#277:

Is it possible to add ways to combine strings that insert characters (like space) as necessary to prevent joining of graphemes between given strings?

No, this idea doesn't work, at least not for the purpose of appending a string of grapheme length N with another of grapheme length M and getting a string that is always N+M graphemes.

Add graphemes<?, graphemes-ci<?, etc. [...]
Add read-grapheme [...]

I haven't needed these, so far, and I haven't tried to add them.

Change port line counting to count positions and columns in graphemes, not characters — either always or by request.

Current implementation is "always".

Change draw-text in non-combining mode to draw graphemes independently, not characters — either always or by request.

Current implementation is "by request".

Change string-snip% to count by graphemes instead of characters — either always or by request.

Changing "item" in the editor's terminology to mean graphemes instead of characters does not seem to be workable. The editor API and uses are too intertwined with Racket strings and characters. (After arriving at that conclusion, I considered again the possibility of changing Racket's notion of "character" to grapheme, but only briefly, because it would be a huge amount of effort for a relatively small payoff.)

Instead, the current implementation adds grapheme counting to the editor classes and offers methods for converting between grapheme and character positions. Old methods continue to work in terms of characters, and there is no guarantee of grapheme integrity; for example, you can programmatically position the caret inside a grapheme and insert characters, even though the default implementation of arrow-key navigation would not let a user do that. Instead of having the editor "just work" when it comes to graphemes, we'll need to apply incremental changes to various clients of the editor API to make them do the right thing.

Regexps: maybe not worth doing anything.

"Not worth it" still sounds right.

[97jaz]

Is it possible to add ways to combine strings that insert characters (like space) as necessary to prevent joining of graphemes between given strings? For example, appending two strings in this ways would produce a string whose grapheme count is the sum of the counts of the appended strings.

There are all sorts contexts where you want to preserve grapheme count, and if it's possible, maybe Rhombus string-formatting things should generally use this view (not a character-based view). For example, the equivalent of error: ~a with a string argument should not cause the space before ~a to be combined with the first part of a string substituted in ~a.

That's a good question. From what I can tell there isn't a code point that extends a cluster on one side and prevents extension on the other. So, for example: "e\u200C\u0301" has only 1 grapheme, because the zero-width non-joiner in the middle extends the cluster started by "e," and then the combining acute accent attaches as it normally would. On the other hand, it you replace the zero-width non-joiner with a zero-width space (\u200B), you get three graphemes, because the space doesn't combine with either the left or the right side.

I found a post where a member of the Swift core team mentions this issue. Here's the relevant bit:

It was always well understood that if strings were treated as sequences of
UnicodeScalars, algorithms such as lexicographicalCompare, elementsEqual,
and reversed would produce nonsense results. Thus, in Swift 1.0, String was
a collection of Character (extended grapheme clusters). During 2.0
development, though, we realized that correct string concatenation could
occasionally merge distinct grapheme clusters at the start and end of combined
strings.

This quirk aside, every aspect of strings-as-collections-of-graphemes appears to
comport perfectly with Unicode. We think the concatenation problem is tolerable,
because the cases where it occurs all represent partially-formed constructs. The
largest class—isolated combining characters such as ◌́ (U+0301 COMBINING ACUTE
ACCENT)—are explicitly called out in the Unicode standard as
“degenerate” or
“defective”. The other
cases—such as a string ending in a zero-width joiner or half of a regional
indicator—appear to be equally transient and unlikely outside of a text editor.

Admitting these cases encourages exploration of grapheme composition and is
consistent with what appears to be an overall Unicode philosophy that “no
special provisions are made to get marginally better behavior for… cases that
never occur in practice.”[2] Furthermore, it seems
unlikely to disturb the semantics of any plausible algorithms. We can handle
these cases by documenting them, explicitly stating that the elements of a
String are an emergent property based on Unicode rules.

So, kind of a ¯\_(ツ)_/¯ response, really. (I think it's a good idea to consider that these "cases that never occur in practice" might well occur if they're intentionally sought by a malicious user.)

[rfindler]

+1 to "(I think it's a good idea to consider that these "cases that never occur in practice" might well occur if they're intentionally sought by a malicious user.)"

[spdegabrielle]

fancy-app – Might be a relatively small detail, but better to sort out earlier instead of later.

What is fancy-app ?

[Metaxal]

fancy-app – Might be a relatively small detail, but better to sort out earlier instead of later.

What is fancy-app ?

https://pkgs.racket-lang.org/package/fancy-app