Purge grimoire

[dpsutton]

@arrdem has deprecated grimoire and is redirecting to clojure docs. So now cider-grimoire yields the following:

I spent a few minutes looking at clojuredocs to see if there's an api and I don't believe there is. We'll need to either swap over to clojuredocs or remove this feature.

I'd hope to see it moved towards Clojure docs. @cursive-ide does this and it is very slick. I really enjoy the in-editor docs and I'll miss the fruits of @arrdem 's work. Thanks so much for the great project and happy endeavors going forward.

[bbatsov]

Seems there's some API as there's an old client for it - https://github.com/dakrone/clojuredocs-client/blob/master/src/cd_client/core.clj Not sure if that works still, but I think it might be a nice idea to create some middleware for dealing with ClojureDocs if we decide to support it. This would also allow us to show examples in docs, etc.

[cursive-ide]

I don't think there's an API, but there is an archive available at: https://clojuredocs.org/clojuredocs-export.json. I just prompt the user at startup to download that, and then check for its existence when showing the doc. I think the archive is updated daily.

[dpsutton]

Thanks @cursive-ide ! I was wondering how you did it. Your implementation is very slick and convenient.

[dpsutton]

Added a comment to an existing request for an API to see what kind of permissions are given/needed for this data. Seems its about 3mb so maybe could just throw it in an edn file and distribute with orchard or cider-nrepl?

[bbatsov]

@dpsutton I agree. That's what we did for see-also.edn in Orchard - it's just an export from clojuredocs we update from time to time. We just need to add a couple of lein/make tasks to simplify those exports.

[bbatsov]

See also clojure-emacs/orchard#60 which links to how we've extracted see-also.edn from ClojureDocs in the past.

[arrdem]

/me waves sadly

[liquidz]

@bbatsov I'm testing to use clojuredocs-export.json in vim-iced.
https://github.com/liquidz/iced-nrepl/blob/feature/clojuredocs/src/iced/nrepl/document.clj

In my implementation, I try to receive json-path and cache the JSON data because it seems to change frequenty unlike see-also.edn.

I can contribute this codes to orchard or cider-nrepl. What do you think about this implementation?

[bbatsov]

@liquidz The implementation seems reasonable to me, it's just a bit disappointing that there's no EDN export and now we need a third-party dep to parse the JSON. This means that we have to do this in cider-nrepl, not orchard, as ideally Orchard should stay deps free (outside of Clojure/ClojureScript that is).

Anyways, I guess that's good enough for now. Thanks for offering to help out!

[liquidz]

@bbatsov Thanks!
I'm confirming about EDN format exporting to clojuredocs.
If there is no plan to export data as EDN format, I'll create a PR to cider-nrepl repository.

[bbatsov]

Great!

If there is no plan to export data as EDN format, I'll create a PR to cider-nrepl repository.

Btw, I've been thinking that we might make this happen ourselves as well, because all we need is some trivial web service deployed to Heroku (or whatever) that just pull json export daily and converts it to EDN. Still, I hope @zk would be able to help us with some EDN export straight from the source.

[liquidz]

@bbatsov

Btw, I've been thinking that we might make this happen ourselves as well, because all we need is some trivial web service deployed to Heroku (or whatever) that just pull json export daily and converts it to EDN.

How about using Netlify?
I made a simple project as a test to deply converted EDN file to Netlify daily.

• https://clojuredocs-edn.netlify.com/export.edn
• https://github.com/liquidz/clojuredocs-export-edn
  • Deploying EDN file daily via schedule configulation in CircleCI
    • https://github.com/liquidz/clojuredocs-export-edn/blob/master/.circleci/config.yml#L42-L43

[bbatsov]

@liquidz Awesome! That's exactly what I had in mind.

Now you can add the functionality straight to Orchard and cider-nrepl will feature just a think wrapper around it.

[bbatsov]

@liquidz I finally started working on this and I noticed that the export can be tweaked a bit to be more convenient - it keeps records of all the edits made by everyone and I think we just need to discard this info and put things like :see-alsos, :notes and :examples as top-level keys of each symbol associated with a vector. We can keep the export raw and have orchard massage the data, but it seems to me we can just clean up the export directly (this is also going to make it much smaller to download).

[liquidz]

@bbatsov I'm not sure what structure you are expecting.
Could you tell me the concrete example?

[bbatsov]

@liquidz Sure!

Here, for instance, I'd drop all the author information about how created/edited something:

:vars [{:ns "clojure.core", :name "def", :type "var", :see-alsos [{:created-at 1289040035000, :author {:login "boxie", :account-source "clojuredocs", :avatar-url "https://www.gravatar.com/avatar/bfc366066e3c1beee98f3a6666728169?r=PG&default=identicon"}, :to-var {:ns "clojure.core", :name "defn", :library-url "https://github.com/clojure/clojure"}, :_id "542692eaf6e94c6970521b15"} {:created-at 1289040039000, :author {:login "boxie", :account-source "clojuredocs", :avatar-url "https://www.gravatar.com/avatar/bfc366066e3c1beee98f3a6666728169?r=PG&default=identicon"}, :to-var {:ns "clojure.core", :name "fn", :library-url "https://github.com/clojure/clojure"}, :_id "542692eaf6e94c6970521b16"} {:created-at 1289040051000, :author {:login "boxie", :account-source "clojuredocs", :avatar-url "https://www.gravatar.com/avatar/bfc366066e3c1beee98f3a6666728169?r=PG&default=identicon"},

This type of data is important for ClojureDocs itself, but it's not important when you just want to show some data in an editor. As I said - it's not a big deal for me tweak the data processing in Orchard, but I'm reasonably sure we won't ever need this type of data so it might be better to make the export more compact. Basically something like this will be ideal:

{:var {:ns ... :name ... :type ... :see-alsos [just a list of vars] :notes [just a list of notes] :examples just a list of examples}

Does this name sense?

[liquidz]

@bbatsov Thanks!
It seems good to me too.
I'll try to put sample data to https://clojuredocs-edn.netlify.com/

[liquidz]

@bbatsov I deployed the compact version here.
https://clojuredocs-edn.netlify.com/export.compact.edn
How about this sample data?

[bbatsov]

@liquidz Awesome! That's exactly what I had in mind! I guess we can update orchard/cider-nrepl to use this data instead.

[wandersoncferreira]

@bbatsov I started to understand the code in cider-clojuredocs and I noticed something with the data you get from orchard/cider-nrepl.

This is the dict output when looking up the symbol def:

(dict "status" ("done") "doc" "Creates and interns a global var with the name
  of symbol in the current namespace (*ns*) or locates such a var if
  it already exists.  If init is supplied, it is evaluated, and the
  root binding of the var is set to the resulting value.  If init is
  not supplied, the root binding of the var is unaffected." "forms-str" "(def symbol doc-string? init?)" "id" "56" "name" "def" "see-also" ("clojure.core/defn" "clojure.core/fn" "clojure.core/defmacro" "clojure.core/defmulti" "clojure.core/defonce" "clojure.core/ns-unmap") "session" "e7babf3b-fe95-49dc-999d-8565bbf9ab72" "special-form" "true" "url" "https://clojure.org/special_forms#def")

It's missing the key ns which is expected by the code in order to make the request through browse-url. I think the fix for this situations would be better at the origin of this data. Can you point some directions to debug further?

[bbatsov]

See the code here https://github.com/clojure-emacs/orchard/blob/master/src/orchard/clojuredocs.clj

[bbatsov]

I assume that simply this is not part of the ClojureDocs export for some reason, which seems a bit weird, but it won't be hard to tweak slightly what we get from it.

[liquidz]

@wandersoncferreira @bbatsov

"special-form" "true"

ClojureDocs export data seems not to have ns key for special forms.
https://clojure.org/reference/special_forms

[bbatsov]

Well, I guess we can just consider the lack of ns as clojure.core then. Or tweak the data a bit more in the exporter service.

[wandersoncferreira]

This assumption is good enough, but I think would be good to tweak the data exporter to include the clojure.core namespace and left the current implementation on cider-clojuredocs unchanged.

[liquidz]

It is easy to tweak exporter :)
https://github.com/liquidz/clojuredocs-export-edn

[liquidz]

@wandersoncferreira @bbatsov
Oh, I misunderstood! Sorry!
It is info op that does not return ns key for a special form.

This is the dict output when looking up the symbol def:

I guess this is a response from info op.

So we should tweak info op or consider the lack of ns as clojure.core.

[bbatsov]

Yeah, we should.

[wandersoncferreira]

I can work on it Thursday. We are sponsoring the Clojure South event here in Brazil and I'm very busy organizing all the stuff we are going to bring to the event. Hope to see you there @bbatsov

[bbatsov]

@wandersoncferreira Likewise!

@liquidz One small note about the export - it'd be really nice if you pretty-printed the export edns, so they are a bit easier to navigate by humans. Not to mention that Emacs handles very poorly super long lines. :D

[liquidz]

@bbatsov OK! I fixed export EDN file.
https://clojuredocs-edn.netlify.com/export.compact.edn

Non pretty-printed version is left here.

[bbatsov]

Great! Thank you! 🙇