Add Linux support by querying the SQLite database directly

[mrjones2014]

https://github.com/sunaku/dasht

https://zealdocs.org/

A Linux client to download and search docsets, just like Dash, but without needing the actual Dash.app since that's a macOS app.

We can probably detect whether we're on macOS or Linux, and use a different backend on Linux. Likely will need a separate Rust backend for Linux.

This will also require tweaking the config a bit I think.

[oblitum]

Zeal is great replacement for Dash on Linux, I think it's actually the more usual solution than dasht. There's also Zeavim. Would be nice to have support for Zeal.

[mrjones2014]

Hmm.. I think it might make sense for that to be a separate Neovim plugin, in that case.

[oblitum]

Okay. Notice that Zeal is a tool for Linux that relies on Dash docsets (in case there's any confusion about them being wholly unrelated).

[mrjones2014]

Ohhh, okay, gotcha @oblitum

relies on Dash docsets

that's what I liked about dasht. If Zeal is the same way, I can work on adding support for Zeal to this plugin then. I've updated the issue description.

[mrjones2014]

@oblitum does Zeal have a CLI? For the Dash integration I'm using the same CLI that powers their Alfred integration.

[oblitum]

@mrjones2014 I never used, my sole usage of it, for years, is just by launching the gui, either directly or from [n]vim with zeavim.

[oblitum]

It should, given there's zeavim.

[mrjones2014]

It seems that's just opening a query, not previewing items. If that's the case we won't be able to implement it as a telescope picker like we can with Dash.

[oblitum]

hmmm, ah okay, didn't thought of the preview feature. Maybe dasht + zeal could work together (dasht for gathering preview data, dasht or zeal for final display).

[mrjones2014]

Opened zealdocs/zeal#1360 to investigate this further.

[mrjones2014]

Dasht seems to have more recent releases

[nogweii]

A path that could be investigated is replacing both Dash and Zeal and search directly in the rust backend; the docsets are just SQLite databases after all. See https://github.com/sunaku/dasht/blob/master/bin/dasht-query-exec for examples on that.

I think reading the documentation with correct rendering and managing the docsets should still be delegated to Zeal/Dash, this plugin doesn't need to be a full alternative to them.

[mrjones2014]

@nogweii woah, did not know about that! I could definitely make that work on Linux that way!

[mrjones2014]

https://rust-lang-nursery.github.io/rust-cookbook/database/sqlite.html

[mrjones2014]

I've started working on this but the big problem I need to solve is: how do I find the location of the actual sqlite database for each docset? On Dash.app for Mac, for example, you can customize in settings the keyword for each docset, so I can't rely on finding it via keyword.

[smithbm2316]

This is most definitely a separate plugin idea, but it would be an interesting to me to just build a plugin that happens to use the Dash docsets, and renders them within a new Neovim Filetype, similarly to how you can directly navigate the Vim help docs natively. I can't speak for everyone, but while I really like Dash.app, I mostly like it because it's one single repository essentially of documentation that is easily searchable. I'd much rather have access to those docs within Neovim itself rather than have to go to a separate app personally, because then I can scroll, search, query, and navigate documentation for anything I want using my Neovim keybindings.

Might be an interesting idea to explore just setting up almost a "Neovim Client" for Dash docsets where you generate the SQLite file with the plugin and add/remove docsets from within the plugin itself. Now that's a lot of just re-doing what Dash.app or Zeal or dasht already do, but it could be a fun plugin idea and give us as the Neovim users much more control of the end user experience. Just kinda spitballing here, it might be something I'd be interested in hacking together myself.

Thanks for all of the awesome work you've done on this plugin so far, it's really excellent! Definitely looking forward to seeing an integration with Zeal or dasht as I'll likely be switching away from Mac back to Linux for work in the near future.

[lukelex]

Does anyone know where the SQLite file lives? I'd love to take a peak at how things are structure there.

[smithbm2316]

Does anyone know where the SQLite file lives? I'd love to take a peak at how things are structure there.

There's a SQLite file afaik for each docset you install. It's not one big file with all the docsets in it. On my machine in the ~/Library/Application Support/Dash folder, here's the Golang docset file structure:

[mrjones2014]

@smithbm2316 I’d be more than open to implementing that here, but the issue I’m running into is that the contents are HTML files.

I don’t know of a good way to render the HTML within Neovim. I’ve tried w3m and browsh (text-based terminal browsers) but neither looks very good.

See #3

[smithbm2316]

I’d be more than open to implementing that here, but the issue I’m running into is that the contents are HTML files. I don’t know of a good way to render the HTML within Neovim.

@mrjones2014 I wonder if there would be an easy way to integrate an html -> markdown converter/compiler and then just render the markdown inside of Neovim. Something like this or pandoc as an optional build step when dash.nvim is installed, but perhaps with a Rust-based package that does that job. Again, very much just thinking out loud here, food for thought though. The more I think about this idea the more I feel like I'm seriously overengineering this in my head 😆 .

I'll have a think about this HTML rendering issue in the coming week, I forgot that the underlying docsets are HTML and not something easily human-readable. I'm unfortunately not versed in Rust at all so I can't say I'd be of much help with the Rust side of things, but if I can think of a way to try and integrate a workflow like the one I mentioned into this plugin using Lua I'd gladly open a PR and try to help. Perhaps what I have envisioned is just not built for Neovim, since the docsets use HTML and are not rendered with Markdown or something else that is human-readable or in a format that is easy to render inside of Neovim natively.

[xbc5]

I've been thinking about this idea for a few months now, and found it because I was looking for exactly this. Being able to just fuzzy find over docs without leaving Neovim is a fantastic idea. I agree with smithb2316, en entirely pure Neovim experience would be great.

I took the liberty of trying out pandoc on a couple of docsets, see here. It mostly works, with some rough edges, but definitely promising.

Some docsets are split up into multiple files, which will need combined. Docsets can also have a slightly different directory structure.

Additionally, perhaps it's also possible to get syntax highlighting for embedded code via Treesitter.

I will watch this closely, and contribute where I feel that I can.

[mrjones2014]

@xbc5 I’m concerned about having to maintain markdown versions of docsets myself. But this idea is more related to #3 anyway, it doesn’t help with fuzzy finding them.

[xbc5]

@mrjones2014

1. couldn't I fuzzy-find over the produced markdown? It might not be plain old text, but is it good enough?
2. you wouldn't have to maintain markdown versions of the docsets, just the translator (i.e. ensuring clean translation) -- users can translate when they install a docset (it's fast);

Are you looking for plain old text? There's code/syntax highlighting to consider, which is most important in this context -- everything else is secondary.

[mrjones2014]

Yeah, but translating from HTML to Markdown is much, much harder than translating from Markdown to HTML. The input space is far greater. Converting HTML to Markdown is far beyond the scope of this project.

[mosaleh52]

@mrjones2014 I have read the discussion and what I have understand that the problem now is rendering HTML inside neovim and the db problems have been resolved ,am I right

[mrjones2014]

@mosaleh52 nope, have not done any technical implementation on this issue. This would involve selecting and auditing a SQLite crate to pull into the Rust backend, setting up queries and database interactions, then taking the results from that and opening the *.html files in the right client based on system (macOS -> Dash.app, Linux -> Zeal or browser).

rendering HTML inside neovim

this isn't something I think we'll be able to do at all. We just want to query the files and open in the right client.

[teto]

there is https://github.com/tami5/sqlite.lua too that could be a native lua implementation

[mrjones2014]

Yeah, we'd want to do all our db interactions from the Rust backend though, otherwise we'd be making two round trips from lua -> Rust -> lua -> Rust -> lua for every db interaction

[mosaleh52]

@mrjones2014 I have basic knowledge writing sql queries , is there any way that i could help
with some direction i will try to find sql related solution

[mrjones2014]

@mosaleh52 You can try figuring out what the data schema looks like for the sqlite databases.

[mosaleh52]

@mrjones2014 you mean for the zeal db installed in my system , am i correct ?

[mrjones2014]

If it’s the same as the Dash docset dbs, then yeah. I think they’re the same but I’m not sure.

[vihu]

Have been following this issue for a while now and was wondering if we could perhaps split it into potentially doable tasks?

Just for information, there have been a few recent interesting developments for neovim support with rust, specifically nvim-oxi and combine that with sqlite.lua bindings maybe we could get the ball rolling?

[mrjones2014]

Yeah I recently discovered nvim-oxi — using it would simplify the existing Rust code base quite a bit, as well as potentially make this task quite a bit easier.

If introducing nvim-oxi I’d be inclined to move all the code to Rust, and use a Rust crate for SQLite instead.

A first step would be adding nvim-oxi, removing mlua, which will probably require a lot of Rust refactoring and rewriting all the Lua code into Rust.

[mrjones2014]

Although additionally, with enhancements to nvim itself, there’s a lot of other things we could do to make the whole plugin a lot simpler. It may be worth doing a ground up rewrite using nvim-oxi and rusqlite

[vihu]

Although additionally, with enhancements to nvim itself, there’s a lot of other things we could do to make the whole plugin a lot simpler. It may be worth doing a ground up rewrite using nvim-oxi and rusqlite

A rewrite from the ground up purely in rust might definitely be worth it. Just one suggestion (from personal XP), instead of rusqlite, sqlx with sqlite feature flag might also serve as an alternative.

Another nice thing about doing it all (from the ground up) in rust is that we could potentially just do the docset binding stuff first and leave nvim bindings entirely (for first pass), nvim-oxi can follow later once we get to that point.

[mrjones2014]

Agreed. I don't have a ton of time to work on this plugin anymore so any PRs are super appreciated.