"Inverse literate" configuration

[shombando]

@nobiot thanks for creating this great package. I wanted to share an example of how I'm using it. I'm using it to create an "inverse literate" configuration for Emacs. Essentially, instead of creating code blocks in an org file and tangling them out, the org file has org-transclusion code blocks to pull in the specific parts of the config.

I wrote a bit more about my motivation behind it with a short example/explanation if anyone is interested. Or you can jump straight to my config and take a look.

I haven't finished writing about all the packages I use yet, but the good thing is that the org file doesn't need to talk every line/block you have in your config only what you want to highlight, which is another "advantage" than a literate config that is tangled out.

Thanks again for the great package, thought I'd share a perhaps unintended use case that is working very well for me!

[nobiot]

@shombando

Thank you for sharing how you use Org-transclusion!
When it comes to literate configuration, I think I'm in the same camp as you. I tried with tangling but I decided that I preferred changing Elisp code directly rather than via source blocks in an Org file and to go back to .el files with comments. Your "inverse literate" makes sense to me.

a perhaps unintended use case that is working very well for me!

I have always thought that if a tool is good, there will be people who put it in good use in ways that its creator has never thought of. So your use case means a lot to me :) Thank you.

On a side note, I really like what you are doing with gemini protocol with your capsule/blog and EmacsVanillaChocolateSwirl. The Emacs wizard idea by Yuan looks also good. I wonder if we/you/I/someone can pick it up from where it was left...

[shombando]

Thanks for the gracious sentiment @nobiot, very kind of you! I went a step further and added an in-buffer function (if that's the right term) to automatically generate the readme.md each time the org file is saved: https://git.sr.ht/~shom/.emacs.custom#exporting-readme-markdown (please let me know if you think there's a better way to do this).

I definitely want to make some time and come back to automating gemini publishing from my single org file that publishes using ox-hugo. As for EmacsVanillaChocolateSwirl it's a (tasty) mouthful of a project name. I have learned a lot more since I started that but I'm really glad I have that "new user perspective" documented. I think I want to convert all of that over to inverse literate, so new users can just hack on the .el file and remove another layer of abstraction while still providing the commentary. What do you think?

I would love for Emacs to have a new user wizard (which only triggers if a config isn't detected) but my emacs-lisp chops are non-existent. But I would love to contribute to an effort like that (I can hack around if there's a structure). If you ever consider it, please let me know.

[devcarbon-com]

I'm planning to use this same approach, and am hoping to make a very streamlined workflow with as little friction as possible. I'll update here when I feel like I have something that is starting to be coherent. :)

[nobiot]

Cool 😎

[shombando]

@devcarbon-com cool, looking forward to seeing what you do and learning from it!

[devcarbon-com]

@nobiot I'm curious, is there a technical limitation that prevents us from using live sync on source code blocks? If it's possible, do you have a rough idea how much effort it will take?

[ParetoOptimalDev]

For totally unrelated reasons I was looking into polymode which somewhat surprisingly might be of help to this problem.

From my understanding it has to differentiate between different types of code within a document. So it replaces org source blocks as I undersand it with:

Spans
span is a contiguous and homogeneous region of text of the same Emacs major mode. There are 4 types of spans: head, body, tail and host.
All spans are closed on the left and open on the right. For example, a span spanning [3, 5) contains character at position 3 but not at position 5. This is consistent with how emacs' functions deal with text properties.

https://polymode.github.io/concepts/

So the polymode implementation has the same requirements for identifying positioning that one would need to transclude org source blocks it seems.

@vspinu If you have the time and interest, can you verify or comment on the above?

Probably getting ahead of myself, but if this is the case and the notion of spans, chunks, etc that polymode have work for both use cases perhaps they belong in emacs core?

[devcarbon-com]

@nobiot Sure. Can you point me to the relevant functions that use *-no-properties so that I can be specific in my conversation with them?

[devcarbon-com]

Ah I think I found it. org-element-src-block-parser and friends.

[devcarbon-com]

Hhm, I think org might have changed such that it works now.

I have a monkey-patched local version of org-transclusion working right now with live-sync on src-blocks.

I'm using org version 9.7-pre, so I don't know for how long the extra properties have been preserved.

The monkey patching was because org-transclusion-beg-mkr and org-transclusion-end-mkr are inclusive of the #begin_src and #end_src markers, so the content length was offset.

[devcarbon-com]

Experimental branch here: https://github.com/devcarbon-com/org-transclusion/tree/feature--org-block-live-sync

[reyman]

Really interesting thread, i'm very interested by such workflow.

In this issue ( #144), i also try to think about the "inverse literate" configuration in a "sharing code source in a team workflow".

When many people use the same repository, with or without org-mode, they change name of function, line number, etc.

Detecting the code source to transclude using generic "uuid" (like org-roam finally) is probably more robust in a team environnement.

<< uuid xcfvdes34d 
def myFunctionA() : 
    print ("helloworld")
    ... 
>>

[devcarbon-com]

@reyman I'm getting closer to releasing a prototype. It's been fun, but I'm also a bit limited in the time I have available to invest in this.

So far I've got (among other things)

• Automatic uuid insertion
• an easy-to-use transient UI
• seamless jumping from code to .org and vise-versa

There's still plenty more on the todo list, and currently it's best tuned for working on lisps.

[ParetoOptimalDev]

Nice job!

I fear if your goal is collaborating with people that update the non-literate code though... I've found any sort of annotation even if minimal makes literate<-->non-literate codebase interaction a non-starter or totally prevents adoption. Or at least makes one very unpopular 😅

However... if you are interested there is a solution that's been used by the leo editor for some time to enable sync between literate/non-literate without any annotations or "sentinels":

The algorithm works with any kind of text file. The algorithm uses only difflib. It knows nothing about the text or its meaning. No parsing is ever done.

Suppose file x has been changed outside of Leo. When Leo reads x it does the following:

Recreates the old version of x without sentinels by writing the @clean x outline into a string, as if it were writing the @clean x outline again.

Recreates all the lines of x with sentinels by writing the @clean x outline into a string, as if it was writing an @file node! Let’s call these lines the old sentinels lines.

Uses difflib.SequenceMatcher to create a set of diffs between the old and new versions of x without sentinels.

Terminology: the diffs tell how to change file a into file b. The actual code uses this terminology: a is set of lines in the old version of x, b is the set of lines in the new version of x.

Creates a set of lines, the new sentinels lines using the old sentinels lines, the a and b lines and the diffs.

This is the magic. Bernhard Mulder’s genius was conceiving that a three-way merge of lines could produce the new outline, with sentinels. The code is in x.propagate_changed_lines and its helpers.

Replaces the @clean tree with the new tree created by reading the new sentinels lines with the @file read logic.

Important: The update algorithm never changes sentinels. It never inserts or deletes nodes. The user is responsible for creating nodes to hold new lines, or for deleting nodes that become empty as the result of deleting lines.

http://leoeditor.com/appendices.html#the-mulder-ream-update-algorithm

[devcarbon-com]

@ParetoOptimalDev, interesting!

I'm not sure how I'd go about implementing such an algorithm for emacs, but I'd be open to suggestions/pull-requests.

The way I'm going about it for my codebase is to try to make the id as unobtrusive as possible.

It looks like this:

(defun marker-at-line (line) ;<id:1672242453>
  "Create a marker at the beginning of LINE in the current buffer."
  (set-marker (make-marker) (line-beginning-position line)))

And it's set to transclude the sexp at point. (or rather, at the beginning of the line excluding white-space.)

[nobiot]

@devcarbon-com I'll be honest and I don't think I can follow the discussion (it's fascinating but the content is beyond me). Is this related to your PR #157? Should we somehow collectively review it?

If this is OK, I'd love others like @ParetoOptimalDev and @reyman to review @devcarbon-com's PR #157.

I am no developer, and I don't think I am very useful for this PR -- the feature is pretty much geared toward developer audience, I think.

Let me know what you think.

Also... As an idea, perhaps @devcarbon-com or someone with interest in the developer usage might like to own this part of the development? (I would love someone like @devcarbon-com with enthusiasm and skills to take the lead). The key functions should be in a single file org-transclusion-src-lines.el (with occasional change to the core org-transclusion.el, whose dicsussion I can also participate in) so the separation of ownership and focus should be feasible, in my opinion. Not sure how this might sound to you -- far-fetched? fair?

[ParetoOptimalDev]

When I get some time I will review #157 further, but a quick glance both the idea and the implementation seem good. I'll comment more in there.

Having someone interested in developer usage owning that part of it sounds like a great idea. I wouldn't be able to because I've not yet been able to integrate org-transclusion into important code in my work or personal life, but that's becoming possible/desireable 😄

[joshcho]

I have taken a keen interest in using transclusion for development workflow. I do full-stack development with Clojure, and I often need to jump between different sets of files. Suppose on the frontend, I have files A, B, C, and on the backend, D, E, F. It's hard to juggle different windows for different kinds of workflows. But with org-transclusion, I can have transcluded A, B, C in a frontend.org file, and simulate the experience of having three vertically stacked windows in a single window. The advantage of transclusion is that "toggling" the visibility of individual files is much easier, not to mention all the org-mode advantages you get.

I think with some :lines trickery you can also change the "view" into the files themselves in order to simulate cursor move in a buffer. A src code live-sync on top of this would be great.