Implement workspaces

[cmyr]

This is a medium-sized project for somebody, which will require a design document and discussion. I am happy to mentor somebody who is interested in taking this on.

Workspaces group a collection of open documents, and enable a variety of features:

• auto-save and session restoration: workspaces will passively track state, including window properties (probably an opaque blob of data updated as needed by the frontend, that might include things like window position, size, and other state) and will also store unsaved changes, the undo stack, etc; a client should be able to pass a 'workspace id' to core on startup, and have be able to easily reopen all previously open documents.

• directory tracking: a workspace should consist of one or more directories, and there should be api for adding and removing directories; these will be used for things like 'quick open'

• (eventually) workspaces should have their own config domain, so that config options can be set on a per-workspace basis

• (eventually) workspaces will provide a new hook into the config and plugin systems, for things like language server support

• multiple workspaces should be possible, and there should be an initial implicit workspace

For an initial version of this, I think it's reasonable to focus on the persistence bits.

[ghost]

Be careful. Some editors that have workspaces make it really inconvenient to edit a file that is not part of a workspace. That's one big reason I switched back from Atom to Emacs, just so I don't have to bother with workspaces.

[cmyr]

Agree, workspaces should be a mostly transparent concept in the general case.

[jansol]

I think the issue is that all the workspace stuff (sidebar etc) should probably be disabled for "I just quickly want to fix up one line here" -like usecases. Also for use as Git's $EDITOR

[akxcv]

Hello. I would love to help with this. Do you think I could work on this feature considering I'm not very experienced at Rust (advanced beginner maybe)? I do understand that this feature is far from simple.

[jansol]

Hello and welcome! The real challenge of this issue is in the design part, hence requirement of a design document. The actual implementation should be doable with a relatively basic understanding of Rust. And the solution does not have to be perfect/particularly elegant at first -- that can be fixed during PR review.

[akxcv]

What's the best way to share and discuss a design document? Should we just use this issue for that or is it preferable to create a PR linked here?

[LiHRaM]

What's the best way to share and discuss a design document? Should we just use this issue for that or is it preferable to create a PR linked here?
@akxcv

Using this issue is fine AFAIK - we tend to do the same on other issues, at least.

EDIT: To clarify - major design decisions, I think, belong here. They can then be followed up on by a PR, but it gives a clear overview of the progress of this feature, which I suspect will be a collection of PR's rather than one big one.

[cmyr]

@akxcv we don't have great processes in place for this right now, but my rough suggestion:

• use this thread to hash out general ideas
• once you have a better sense of what the actual work will be, open a new issue that offers an overview of the design you're thinking of, and describes (roughly) the steps you expect to take to get there. For an example, maybe look at something like [RFC] standardize reporting of async errors #543?

[jansol]

* once you have a better sense of what the actual work will be, open a new issue that offers an overview of the design you're thinking of, and describes (roughly) the steps you expect to take to get there. For an example, maybe look at something like #543?

(or, if that's more up your alley, a pull request -- we are not too picky about the distinction for now)

[akxcv]

Okay, so what are the main questions right now that we need to answer?

I've come up with these so far (this is not very structured for now):

Q: What data belongs to a workspace and how to structure it?

A: We could differentiate between two types of workspace data:

• Workspace config ($CONFIG_PATH/workspaces.xiconfig) where workspace name, path, and editor config overrides can be specified. An example would be:

[xi-editor]
path = "~/Projects/xi-editor"

[a-rust-app-that-uses-two-spaces-for-some-reason]
path = "~/Projects/a-rust-app-that-uses-two-spaces-for-some-reason"
tab_size = 2 # maybe editor config overrides should be nested in a namespace?

xi-core is responsible for watching for workspace config file changes, just as with other config files in this directory. Modifying the workspace root path while the workspace is open should not be a problem.

• Workspace metadata (or state) includes a list of open files (including unsaved ones and files not included in the workspace), state for each file (cursor position, view-specific settings such as tab size, undo stack), and frontend-specific arbitrary data (for mac, this might include window size and position, sidebar state such as open subdirectories).
  Metadata should be updated on almost every editor action (insert/remove a character, open a file, etc.). Not sure about the strategy to persist metadata to disk, though (maybe this should be debounced so that we don't write to disk so much?)

Q: How will the user manage workspaces?

A: First of all, we should allow the user to open a directory just as they can currently open a file. Opening a directory will create an "implicit workspace", which is a temporary workspace that doesn't have a name or an entry in the config file. (Question here: should implicit workspace metadata be stored on disk? I imagine it should as this will enable the user to restore a lost session even if they didn't "create" the workspace explicitly.)

1. Creating a workspace:
   • Adding an entry to $CONFIG_PATH/workspaces.xiconfig will create an empty workspace. This will happen either when the editor starts, or when the config gets modified. (A question arises: what happens to workspace metadata when the user renames/removes a workspace or changes its path?)
   • Saving a currently open implicit workspace. This will modify $CONFIG_PATH/workspaces.xiconfig.
2. Opening a workspace:
   • Opening a directory which belongs to a workspace (should it include subdirectories?)
   • Opening a workspace by its name from the frontend
3. Removing a workspace: ???, as specified above
4. Editing workspace configuration:
   • Changing the workspace name: ???, as specified above
   • Changing the workspace path: ???, as specified above
   • Modifying editor config overrides for the workspace: same as changing any other config files

Q: What should the protocol look like?

A: This is not clear right now, but there surely should be methods for creating and opening workspaces as well as updating workspace metadata. The core should also notify the frontend about changes in workspace directory tree. Also, some parts of the current protocol might need to be modified.

Notes:

I think the issue is that all the workspace stuff (sidebar etc) should probably be disabled for "I just quickly want to fix up one line here" -like use cases.

I think this is as simple as "show the sidebar if a directory is open, and don't show it if only a single file is open".

That's all I remember thinking about for now.

[LiHRaM]

Great input!

Q. How will the user manage workspaces?

...
2. Opening a workspace:

I've been really frustrated with how most editors handle language-based project files (think Rust .toml files, C# .sln or .csproj files, etc.) - I think we should provide some way for Xi-Editor to just open these files as workspaces. The implementation of the different types of files should be provided by plugins, but they should be able to use API exposed by the workspace implementation in core.

So this touches on both protocol in core and possibly some plugin API.
Instead of modifying the theoretical $CONFIG_PATH/workspaces.xiconfig, the API should also provide a way for plugins to modify the project file located in the workspace directly. This could probably just be handled by a flag in the workspace API (xi-workspace vs. external workspace-type, or something?)

I think this is as simple as "show the sidebar if a directory is open, and don't show it if only a single file is open".

Agreed, that's a discussion for the front-end implementation.

[akxcv]

Personally, I've always preferred to just open directories, not project files, it just feels more intuitive. But yes, it makes sense to also support project files, although it might get tricky in particular multi-language setups. There's also a problem (in VScode at least) when dealing with a monorepo where Rust integration doesn't properly work if Cargo.toml is not in the root of the workspace (not sure if this is related to our discussion, though).

Speaking of editors like VScode and Sublime... Those editors (and Atom, I believe) only implement "implicit workspaces" natively, and you can install a "project manager" plugin to select and name workspaces. So maybe we should consider taking this route as well, at least initially.

[akxcv]

So I think we need to come up with something as simple as possible for the initial implementation.

Let's implement implicit workspaces only. No names or configs.

Workspace data

• A set of states for each open file (cursor position, language, tab size, etc).
• Metadata: A file tree of workspace directory + frontend-specific data

This needs to be described in detail.

Data gets updated on text edit operations, file open/close operations, view config changes. We will not store the undo stack yet (#897). The core should also watch for changes in the workspace directory and update the file tree accordingly.

Workspace metadata should contain arbitrary frontend-specific data. An example of frontend data xi-mac could be window width and height, or sidebar state (open dirs). This means we should allow each frontend to have its own specific spot for storing its metadata.

Workspace API

• Create a directory/file
• Move a directory/file (includes rename)
• Delete a directory/file

Hmm, surely, the API will have to include smth else as well

Open questions

• How and where do we actually store workspace metadata?
• Should we think about traversing directories lazily straight away? (e.x. not watching/storing files of subdirectories that haven't been used). This would complicate things greatly, but will allow us to use xi-editor on projects with dirs such as node_modules.

[cmyr]

Okay, so I think the way I've been thinking about this is really two separate and maybe only tangentially related issues:

state restoration

This has two goals: preventing data loss and allowing the user to quit and reopen their client without losing their open documents and window positions, etc.

This is not a small feature! Off the top of my head it requires:

• keeping track of the absolute delta between the current buffer state and the last saved state, for each open buffer; this is distinct from the undo stack (the undo stack can contain a lot of additonal information, but is also bounded in length; there is no guarantee that the undo stack provides enough information to recreate the document)
• keeping track of the view state (the selections and the viewport position) for all views, and making all of this data serializable
• keeping track of undo state; this is non trivial because it is currently tied up in the general CRDT state, which we don't want to try and serialize/load on the fly; instead we want to serialize some summary of each undo op on the stack; this should probably be a follow-up project
• keep track of any user-overridden config settings;
• providing an API for the client to pass us arbitrary state that it will need in order to restore the session, such as the positions of windows and whether they're displaying a sidebar or anything else a particular client might provide
• persisting this on the fly, efficiently and performantly
• restoring this state, quickly, when needed;
• probably some other stuff I'm forgetting?

This is closely related to this xi-mac issue, about implementing split views; basically xi-mac currently relies on a bunch of Cocoa APIs for things like state restoration, but those make it harder to do other things, so we'd like to ditch them. In this case, xi-mac will rely on this work.

Separately, there is the question of grouping related files and opening folders. These are related in my head because I think that a 'workspace' object in core is the natural place to handle the persistence stuff above, but I might be stretching. In any case:

workspace folders

This is probably the smaller of the two features, although it might have some large implications in terms of API and how we architect core. In any case, the basic functionality it provides (I think) is the ability to group multiple files and folders together into a new scope.

A workspace, in this context, is then just a set of files and folders; it's main purpose is to provide scope for certain features (quick-open might be scoped to a workspace, for instance) as well as to provide additional information to clients that might want to show a file browser or related. Something similar with plugins.

There are some API options:

• we could reuse the existing new_view RPC, and give it an (optional?) 'workspace' field; if we also allow new_view's path argument to take directories as well as files, this feels a bit overloaded, but seems to give us most of what we want.
• It also isn't clear that we always want to be opening things to add them to a workspace; so we could instead just have explicit 'add_to_workspace' and 'remove_from_workspace' RPCs.

There are other questions I have; can you have a workspace that doesn't contain a directory, or is that meaningless? Can you add files to a workspace, or only directories? If you're just opening an editing files, does each one get its own implicit workspace, does it have no workspace, or are they all part of one big implicit workspace?

So: I think we need to figure out exactly what we're building first, and then break those tasks down into reasonable sized chunks. I think it might make sense to move the 'state restoration' stuff above into its own issue?

[akxcv]

Not sure about separating the two issues quite yet, I think we need to come up with a plan first.

My thinking is that we should treat everything as a workspace. If a single file is opened, it's in a workspace. If a directory is opened, it's also in a workspace. When we just open xi-editor, there's an empty workspace with a single empty view. I think that this way our model will be much clearer than having multiple different cases.

I also don't think new_view should take directories, as I imagine this RPC to only be responsible for opening files. If we agree to treat everything as a workspace, the workspace_id field will be mandatory here.

About state restoration, I think a lot of implementation details for this depends on what we decide to do with workspace folders. Again, if we treat everything as a workspace, this becomes simpler, as we'll come up with a unified model of "state restoration for workspaces".

[ghost]

Trying to understand what you mean with workspaces and why the Xi core should even have them is very confusing. From this thread I can make a lot of interpretations. For example according to one post, workspaces are a way to write INI files to tell Xi the tab size for use in Rust files, something no one would ever need to do. I wonder if there's an actual use case or if people are just trying to design something for the sake of designing something, which isn't an effective way to produce meaningful software. Debating something that doesn't have a defined purpose is also not productive.

Having some mechanism to allow persistence would be very good, and I think this shouldn't be mixed up with workspaces. I really doubt it would be good for the Xi core to have any features for grouping files into workspace at all. A frontend may group opened files by workspace, with a workspace having some plain text files using the Xi core, some PNG files using another backend, some web views using yet another backend and some Interface Builder / Glade / whatever files using another completely unrelated backend. Putting the workspace features in the backend for plain text files isn't any good. The frontend can better manage workspaces without help from Xi. Xi just needs to persist things on a per file basis, and the frontend will group the files into workspaces, together with other file types using other backends, and other frontend settings that Xi doesn't know about and shouldn't ever know about.

[akxcv]

We're exploring here, so it's only natural to see conflicting suggestions in this thread.

Maybe you are right about the core not knowing anything about workspaces, but who should be responsible for watching the workspace directory, for example? Like when a git pull happens and new files are added. Imagine also that files might be on a different machine than the one the frontend is running on... Please do correct me if I'm wrong about these concerns.

[ghost]

@akxcv Who should be responsible for watching directories is a good question. Easiest would be to build it directly into the frontend. Another solution that may be worth exploring would be to create another file-type-agnostic watch-directory-backend, where you can plug in Xi to handle plain text files (and in the future possibly rich text files) as well as other backends to handle other file types.

[jansol]

I view "workspaces" as an amalgamation of "projects" and "groups". Because of that I really like the tag-based approach suggested in #374. That way we could group files opened together, by opening a directory or via the file browser sidebar, while not assigning a "workspace" tag to files opened by themselves. Additionally there could be plugins that find and parse project files for $toolset and tag files accordingly.

This way the core would only need to support generic tagging of buffers, which has other useful applications (scoping for search/plugins/refactoring, likely tons of more creative things that I haven't thought of). Files opened by themselves would not get a workspace tag, disabling all project-related plugins for a smoother experience.

Two questions that I didn't see mentioned:

• How should multiple workspaces be handled? Can plugins require support for multiple instances of themselves for separate projects? Or do we require them to deal with multiple simultaneous workspaces how they best see fit?
  • Buffer tags should be known to plugins anyway so it seems feasible to require plugins to deal with that.
• Tags on files that are not currently open but are part of a workspace. Maybe keep a listing of files that are "known" to the "working set" for tagging purposes in core?
  • Needed at least for features like "Find in project" and refactoring, both of which are very nice to have support for even if they are not directly done by the core.
  • Plugins should be able to add/remove files to that set (e.g. from a project file or by walking a directory tree). Whoever creates the tag would be responsible for updating the list of buffers with that tag.
    • Watching things could be a core service that's exposed to plugins so they can register interesting files (e.g. the project file) or directories and get notified of changes.

Personally I think the tags approach would be a very simple solution that is both powerful enough for IDE-level features and flexible enough to fold everything out of the way when it's not wanted.

[jansol]

Regarding earlier discussion in this thread:

Adding an entry to $CONFIG_PATH/workspaces.xiconfig will create an empty workspace.

A global list of all workspaces will grow boundlessly for no good reason. If xi wanted to be an IDE with full-blown multi-project management and "recent projects" (á la xcode's launch window) then this might make sense, but for "just" a text editor it IMHO doesn't. There may also be privacy arguments if you e.g. open a confidential remote project on a shared computer and forget to remove it from the workspace list.

For things like per-project tab config there are already solutions that don't require global state from us. Editorconfig lines could be parsed by plugins or the core. Things like rustfmt.toml or .clang-format can be parsed by their respective language/toolchain plugins.

who should be responsible for watching the workspace directory, for example? Like when a git pull happens and new files are added.

I'd say that watching files and managing the workspace could reasonably be separate tasks. The former makes sense to do in core since it's already watching open files as well as its own config file. The latter should be the responsibility of whatever plugin manages the workspace. Note that a workspace may or may not be related to directory hierarchies -- I have a tmp folder with various single- or two-file projects jumbled together and it would be useful to be able to just open a handful of related files and have them act as an ad-hoc workspace.

Q. How will the user manage workspaces?

The same way they normally do. Be that via visual studio or by updating CMakeLists.txt. The corresponding plugin is in charge of updating the watchlist based on whatever they used to generate the workspace. We could have support for adding/removing/renaming files in projects but that requires cooperation from the project plugins. The core does not really need to know about it.

EDIT: non-project workspaces would be just drag&drop and "add/remove files"

[akxcv]

Adding an entry to $CONFIG_PATH/workspaces.xiconfig will create an empty workspace.

Q. How will the user manage workspaces?

Yeah, it's clear to me now that this is currently out of scope for now.

I'd say that watching files and managing the workspace could reasonably be separate tasks.

Yeah, I agree. Watching files could be a responsibility of a plugin; the core probably shouldn't concern itself with that.

[jansol]

Closing this as per roadmap discussion on IRC. TL;DR: The concept of "workspaces" is too hazy at the moment and it is possible to get a decent experience (including things like LSP) without really defining it, so let's do that first and then think again if workspaces still feel necessary.