I'm finding it difficult to work with unist's {[k: string]: unknown} catch all index for Node. Wipes out a lot of typechecking.

Using a version of Node without the catch all really helps for internal typechecking - but anything that uses unist then needs to have types switched to e.g., Document & unist.Node, which is a bit annoying.

Trying to figure out the nicest way to handle this that gives us good type safety but doesn't require external API changes.

Specifically, the issue arises when using functions from unified which expect e.g., Document to support arbitrary keys. Not sure why it's been handled like this, as the functions in unified shouldn't be using any keys they haven't verified exist through typing (and thus the typing in unist shouldn't need to require support for arbitrary keys).

Re Node, I've opened a discussion, but I don't know whether the changes will be agreed or not.

For now, I'll keep thinking about how we can have good type safety internally, whilst allowing users to choose the less safe variant of Node from @types/unist when they prefer. I'll do this in a separate PR to make it easier to review, then either split this PR to not include the changes, or rebase it onto that PR @xiaoxinghu.

@xiaoxinghu Okay, I have a good solution (at least internally). (Edit: still trying to fix)

As before, internally make sure to import Node and Parent from types.ts rather than unist.

Now, when exporting in index.ts we just add the index signature to Node (see here), and done! That's all we need to maintain compatability 😄

Edit: maybe I spoke too soon---this seems to be causing issues too (needed to clean and recompile). Gonna see if I can get the re-defining of the interface to work

Edit: the fix didn't work - propagated too quickly. However, discussions are going well so we might be able to use Node from unist if it gets updated :)

Edit: unist is updated! Hopefully will be able to propagate this to the other orga packages.

@xiaoxinghu update on block parsing.

There are three main cases of block contents we need to watch out for:

• "greater" and "special" blocks, which can contain other elements
• "element" blocks, which can contain only text
• "verse" blocks (a special case of "element" blocks), which can contain objects (but not elements)

I've updated the parser/lexer so it can now handle "greater" and "element" blocks, and to a degree it can handle "verse" blocks.

However, as it currently stands, I'm needing to add exceptions for lexing objects for the verse block in a way that we will likely need to be able to apply in other areas, but currently isn't very maintainable.

As a result, I'm looking into one of these two methods:

1. rewriting parts of the lexer so that we can have a more state/context-based system - i.e., we only lex certain tokens when we know they have correct meaning (can only lex object tokens in verse block, for example)
2. simplifying the lexer to just look for simple tokens, and do more of the work in the parser

Point 1 has the advantage of simplifying some of the parsing, and means we don't have to worry about converting tokens in the parser.

Point 2 has the advantage of simplifying lexing to be more clearly token-based without too much lookahead, but means that we may need to do some re-parsing depending on how we handle the lexer.

I think point 2 currently feels a bit more daunting beacuse the lexer needs to be more atomic - once the lexer is more atomic then it should actually be easier.

I'll look into point 2 and let you know how it goes - it should make life a lot easier in updating parsing in the future.

👍 Wow, what a refactor!

👍 Wow, what a refactor!

Yep. I'm trying to be as comprehensive with tests as possible to make it easier to review, but it's always gonna be a bit of a beast.

Once the backend stuff is done w/ the lexer and parser, I'll need to make sure that all the frontend packages work correctly with this, which might be harder.

@GuiltyDolphin , this looks dangerous to me, I suggest that we break these changes down into smaller pieces, because I am doing some refactor work myself which is pretty major. Also, I am changing the way tests are written, so I think it'd be easier to do these tests after mine, I will try my best to get it out of my computer.

@xiaoxinghu Alright! I'll close this for now until your changes are done, and then see how things look after that.