Multi-IDE Features for 2.9

[samuel-williams-da]

Note that this is on main despite being 2.9 features, as all these features are needed in 3 as well.
This PR, and all multi-ide PRs before it will need backporting to the main-2.x branch.

TODO:

• Rework debug logging into log levels
• Restart Multi-IDE automatically, instead of vscode window reload
• Make the vscode options less experimental
• Pass down command line flags (telegraphy, extra args, etc.)
• Change dar deps path to be shorter
• onChunks running out of input doesn't need to be a warning
• Test on windows
• Test daml files with no daml.yaml error correctly
  • Errors over LSP as expected, though UX isn't great, as vscode gets mad
• Check if --experimental should still exist
  • Unrecognised even with the previous IDE.

Content

This is a very dense PR, intended to be reviewed in a call, commit by commit.
Following are the features implemented:

Dar Unpacking

When we go to definition on a package that isn't listed in the multi-package.yaml, we now unpack the source code from the dar, recreate the package structure, and send the user there (where we will also spin up a new subIDE)
The dars that the multi-ide knows about, and can unpack for jumping, are the union of all dependencies and data-dependencies of every package listed in the multi-package.yaml, as well as all dars listed in the dars field added to the multi-package.yaml. This fields current only purpose is providing information to the IDE (source code for transitive dependencies), but can later be re-used for package name imports.
Caveats:

• We do not have the original build options of a dar, and as such, cannot guarantee it will compile (e.g. someone disabled some warnings to build it)
• We do not have the original direct dependency set for a dar, so if a transitive dependency has the same module path, we will hit clashes on unpacking

Both of these issues would be solved by adding a daml.yaml derivative file to dars.

Packageless ides

When opening a new file in VSCode using ctrl + n, and setting that file language to daml, previously the multi-ide would crash, as it did not know how to spin up an IDE for a file that doesn't belong to a package.
We now create a bare minimum IDE (containing only prim + stdlib) in a temp folder, and direct those requests to it.

Hot-reloading

Changes to daml.yaml, multi-package.yaml and dar files will now shutdown (and later reboot, see error recovery) any IDEs related to that file, and update any multi-ide state.
This allows for changes like adding dependencies, renaming, changing build options, etc.

Error recovery

This is an extension of hot-reloading, which makes some tweaks to the reloading logic itself, but mostly handles all the error states I could think of. This includes:

• SubIDE fails to start
• SubIDE crashes while running
• Multi-IDE cannot read multi-package.yaml, daml.yaml, or dars it knows of

In the case where a SubIDE fails, it is immediately attempted to be rebooted. Any pending messages the previous instance had are resent, in an attempt to seamlessly recover for the user.
If the reboot fails, all pending messages are answered with empty replies, all open files relating to that IDE (.daml + daml.yaml) have a top level diagnostic showing the SubIDE error. The subIDE will then refuse to reboot until either the daml.yaml or dars are saved.
When the multi-ide cannot be parsed, similarly the error is shown at the top of the file, no multi-ide state will be stored (so no jump to definition cross package), and it will not be reloaded until it is saved.

[samuel-williams-da]

Some notes from group review of first commit

[samuel-williams-da]

Note: unpacking dar cannot work for dars that use module prefixes, as we are missing this information in the dar.

[moisesackerman-da]

Note: unpacking dar cannot work for dars that use module prefixes, as we are missing this information in the dar.

sad_trombone.wav

[samuel-williams-da]

Noticed that the removal of "old" shutdown requests when creating a new subIDE sometimes happens before the response comes through for a shutting down SubIDE, leads to the shutting down IDE sending a response that the multi-ide doesn't recognise. This crashes the multi-ide, which then gets spun back up by VSCode, and everything "recovers", which is why I didn't notice in testing. Happens randomly when rebooting (from daml.yaml or dar save)
Should be fixed

[samuel-williams-da]

Current windows issues:

• daml-ghcide doesn't shutdown when sent exit message (issue existed before multi-ide), extension got around this by killing the process directly with SIGTERM. We don't want to do this, as it causes issues for handles.
• Flush of handle to subIDE that shutdowns early blocks indefinitely - probably fixed by disabling buffering

[samuel-williams-da]

All above issues fixed with last commit, onto reviews!

[samuel-williams-da]

TODO: Ticket for language server sometimes not shutting down on windows. Typescripts languageClient.stop() does reliably shut it down when called directly, but for some reason, when in deactivate or as a disposable, it often (~50-70% of the time) doesn't shutdown, leaves processes using ~100mb per subIDE

[dylant-da]

LGTM! Thanks for the great work, this is definitely very complex and a lot to manage.

[dylant-da]

Do we need these var synonyms? A quick search doesn't turn anything up for them being used outside the MultiIdeState.

[samuel-williams-da]

I like having them as it gives me an opportunity to name and comment them separately. Perhaps they could be newtypes to provide more value, but I think dropping them and moving all this into the MultiIdeState would make it quite painful to read

[samuel-williams-da]

Maybe InitParams could go, but its adhering the content + tvar pattern I have going

[dylant-da]

Should we be worried about how this will interact with, say, a deeply nested complex folder that needs to be searched? E.g. a node_modules folder.

[samuel-williams-da]

Its a fair point, though this logic is handled by VSCode, which one would hope is implemented efficiently.
I'm not sure what could even be done here through, bar writing patterns with explicit exclusions of some directories - if glob support thats - and even then I don't think that should be the job of the language server to handle

[dylant-da]

If we're passing unrecognized arguments from multiide down to sub ides, those subides should be able to cope with unrecognized arguments too - can we add the forwardOptions modifier to cmdIde as well? Maybe emitting a warning from IDE so that unrecognized arguments can be debugged.

[samuel-williams-da]

I believe the ide is already running with a lax parser, but I'll check

[samuel-williams-da]

It's not, they do error but the user is informed elegantly. I feel like it's probably valuable to tell the user when these don't work, rather than silently ignore them. For anything package specific, they should live in the daml.yaml build-options

[samuel-williams-da]

I'm going to handle improving the situation for daml files without a daml.yaml, alongside better handling for creating/deleting daml.yaml files (regarding the openFiles set) in another PR.
I've planned out whats needed, and its more significant than a couple lines

[dylant-da]

I'm happy with the parts you decided to keep the same